Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Support HTTP sender #1383

Merged
merged 1 commit into from
Apr 8, 2024
Merged

feat: Support HTTP sender #1383

merged 1 commit into from
Apr 8, 2024

Conversation

fangyinc
Copy link
Collaborator

@fangyinc fangyinc commented Apr 7, 2024

Description

  • New HTTP sender operator.
  • New network command for debugging. DB-GPT not support global proxy for #616. Here is a solution to the problem via port forwarding + proxy.

How Has This Been Tested?

  1. make test
  2. Test forward.
dbgpt net forward --local-port 5010 \
--remote-host api.openai.com \
--remote-port 443 \
--ssl \
--proxies http://127.0.0.1:7890 \
--timeout 30

Then connect to it.

export OPENAI_API_BASE=http://127.0.0.1:5010/v1

curl $OPENAI_API_BASE/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "hello"
      }
    ],
    "stream": true
  }'

Snapshots:

Include snapshots for easier review.

Checklist:

  • My code follows the style guidelines of this project
  • I have already rebased the commits and make the commit message conform to the project standard.
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Any dependent changes have been merged and published in downstream modules

@github-actions github-actions bot added the enhancement New feature or request label Apr 7, 2024
@csunny
Copy link
Collaborator

csunny commented Apr 7, 2024

It does not work for me. When I test use above command, the error occur, 500 Internal Server Error

(dbgpt_env) magic@B-4TMH9N3X-2120 DB-GPT % dbgpt net forward --local-port 5010 \
--remote-host api.chatanywhere.tech/v1 \
--remote-port 443 \
--ssl \
--proxies http://127.0.0.1:7890 \
--timeout 30
INFO:     Started server process [16966]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5010 (Press CTRL+C to quit)
Forwarding request to https://api.chatanywhere.tech/v1:443/v1/chat/completions
INFO:     127.0.0.1:63476 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [16966]

@fangyinc
Copy link
Collaborator Author

fangyinc commented Apr 7, 2024

It does not work for me. When I test use above command, the error occur, 500 Internal Server Error

(dbgpt_env) magic@B-4TMH9N3X-2120 DB-GPT % dbgpt net forward --local-port 5010 \
--remote-host api.chatanywhere.tech/v1 \
--remote-port 443 \
--ssl \
--proxies http://127.0.0.1:7890 \
--timeout 30
INFO:     Started server process [16966]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5010 (Press CTRL+C to quit)
Forwarding request to https://api.chatanywhere.tech/v1:443/v1/chat/completions
INFO:     127.0.0.1:63476 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [16966]

Please replace --proxies http://127.0.0.1:7890 with your local HTTP proxy, or delete it.

@csunny
Copy link
Collaborator

csunny commented Apr 7, 2024

The server only print this warning, but i have already installed websockets. 🤔

WARNING:  Unsupported upgrade request.
WARNING:  No supported WebSocket library detected. Please use "pip install 'uvicorn[standard]'", or install 'websockets' or 'wsproto' manually.
WARNING:  Invalid HTTP request received.

@Aries-ckt
Copy link
Collaborator

image

csunny
csunny approved these changes Apr 8, 2024
View reviewed changes
Copy link
Collaborator

@csunny csunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+

Start server

dbgpt net forward --local-port 5010 \
--remote-host api.chatanywhere.tech \
--remote-port 443 \
--ssl \
--timeout 30
INFO:     Started server process [42131]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5010 (Press CTRL+C to quit)
Forwarding request to https://api.chatanywhere.tech:443/v1/chat/completions
INFO:     127.0.0.1:60158 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Request

export OPENAI_API_BASE=http://127.0.0.1:5010/v1

export OPENAI_API_KEY=sk-xxx

curl $OPENAI_API_BASE/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "hello"
      }
    ],
    "stream": true
  }'

Response

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" How"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" can"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" I"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" assist"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":" today"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{"content":"?"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Bbh2eIZo5T93ep3dEfy5PyplBDvd","object":"chat.completion.chunk","created":1712553528,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_592ef5907d","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}

Aries-ckt
Aries-ckt approved these changes Apr 8, 2024
View reviewed changes
Copy link
Collaborator

@Aries-ckt Aries-ckt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+

@Aries-ckt Aries-ckt merged commit 634e62c into eosphoros-ai:main Apr 8, 2024
7 checks passed
@fangyinc fangyinc mentioned this pull request Apr 16, 2024
Closed
15 tasks
Hopshine pushed a commit to Hopshine/DB-GPT that referenced this pull request Sep 10, 2024
@fangyinc @Hopshine
feat: Support HTTP sender (eosphoros-ai#1383)
597c371
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@csunny csunny csunny approved these changes

@Aries-ckt Aries-ckt Aries-ckt approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fangyinc @csunny @Aries-ckt